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Objectives 


First in the World: Promoting Active Learning Strategies Through the Flipped Classroom 
Model is a project funded by the U.S. Department of Education. It partners San José State 
University, California State University, Los Angeles, and California Polytechnic, Pomona with 
the nonprofit educational agency WestEd. The project is developing and studying the impact of 
workshops and professional Flipped Learning Communities (FLCs) offered to science, 
technology, engineering, and mathematic (STEM) faculty in support of “flipping” instruction. 
One goal of the project is to complete a randomized controlled trial study to examine what 
works, for whom, under what conditions in first semester college calculus. 


Perspective 


There are many ways to "flip" a course. In "flipped" teaching and learning the grounding idea is 
to switch the traditional uses of in- and out-of-class learning time. Ina flipped course, students 
spend time on activities with lower cognitive demand (e.g., information gathering and review, 
basic skill building) outside of class. This individual space effort occurs before and after 
students attend class meetings. The focus in group space (1.e., class meetings) is more complex 
and higher cognitive demand activity, with students doing cooperative or collaborative tasks 
that leverage the immediate availability of the instructor as expert consultant and guide (e.g., 
group-worthy tasks, inquiry-based learning, team-based tasks, labs). Clear from the current 
research is that such an approach may result in better outcomes for students, both in terms of 
achievement and quality of learning experience (Deslauriers, Schelew, & Wieman, 2011; Hake, 
1998; Laursen, Hassi, Kogan, & Weston, 2014; Lord & Camacho, 2007; Maciejewski, 2016; 
Prince, 2004; Ruppert, 2003; Walczyk & Ramsey, 2003). However, there never has been a 
rigorous test of the effectiveness of flipped instruction for improving learning in college 
calculus in the United States. 


Though considered the gold standard for U.S. research funding, randomized controlled trials 
(RCTs) in authentic educational research settings are problematic. Such studies depend on 
"balancing research ambition against operational reality" (Gueron, 2002, p. 19) in recruiting of 
teachers and/or students to a study and in obtaining data, particularly personally identifiable 
data. While some candid reporting on recruitment in K-12 settings has been offered (e.g., 
Gueron, 2002; Roschelle, et al., 2016), little has been shared openly about the stumbles and 
struggles encountered when researchers attempt to implement idealized designs in the wilds of 
U.S. colleges and universities. One recent addition to the literature has offered insight into what 
can support success in recruiting and retention of post-secondary students (Terrell & Bugler, 
2018). The work reported here is focused on recruiting and retention of post-secondary 
instructors in an RCT. 


Methods 
The Plan 


The flipped calculus study is an RCT across first semester calculus classes at the three large 
state universities in the project. When funded, the original design of the study was vetted and 
approved by the U. S. Department of Education's First in the World evaluation staff, who relied 
on standards asserted by the then-current version of the What Works Clearinghouse handbook 
(WWC, 2014). Guided by initial power analyses, the design called for 20 instructors to 
participate for two semesters, each teaching two sections of calculus each semester, with about 
25 students per section (i.e., a total of about 2,000 students across the two semesters). Each 
semester, instructors would be their own comparisons (1.e., teach one section flipped and one as 
usual). Each faculty member would have pre- and in-semester professional support for flipping 
one class section. The project would collect data both semesters, though it was acknowledged 
that the project-designed professional supports for instructors to learn to "Do the Flip!" were 
being (re)designed throughout the first semester of the study. 


The Reality 


In any given semester, a total of 20 to 25 sections of Calculus I exist across the three 
universities. So, virtually all calculus instructors would have to participate. At the same time, 
math department chairs are constrained by many conditions, including the realities of academic 
freedom and faculty union rules about making assignments to teach calculus. Thus, we could 
not ensure that participating instructors would be assigned two sections for two semesters in a 
row. Involving one or more additional universities was beyond the funding available. As a 
result, prior to recruiting, the design was revised to have a "Plan B" for those cases where 
instructors were interested in participating but were assigned just one section of calculus. The 
revised plan, also vetted and approved based on WWC (2014) standards, led to the design 
shown in Figure 1. Group 1 includes those eligible in the original design and Group 2 
represents Plan B, for those teaching just one section of calculus in a semester. 


Design Visualization — Each Semester 
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Figure 1. Original (Group 1) and additional option (Group 2) for participation. 


Compounding the challenges of recruitment was the fact that one of the three colleges was 
delayed in a planned move to the semester system. Hence, their faculty could not participate at 
the same time as faculty at the other two colleges. Most calculus courses at the three sites are 
taught by conditional faculty (e.g., non-tenure track or adjunct instructors). As of this writing, 
the calculus study is in its first semester at the two colleges ready for it. Of the 10 instructors 
originally recruited, 2 were reassigned before randomization of sections, 1 dropped before 
randomization of sections, and 7 are active participants. Reporting on communication with the 
local leaders of the professional development for faculty and on retention of instructors in the 
study are the topics of a planned, future, report. 


Data Sources and Data Collection 


Data sources include a calculus readiness pre-test and a calculus concepts post-test to be 
completed by students in all class sections, regular teaching logs by instructors, surveys 
completed by instructors about their mathematics instructional experiences before and during 
each semester of the study, an end-of-term survey by students about their experiences in their 
calculus course, and anonymized transcript data from each college's Institutional Research 
office about student preparation, course grades, subsequent course-taking and course success, 
and demographics. 


Results (to date) 
Administrators 


We found that more often than not, STEM leaders in the universities had limited familiarity 
with the institutional review board (IRB) role in conducting such a study and the protections 
offered to participating faculty as human subjects in research. To avoid having faculty being 
"voluntold" to participate in it - a violation of IRB and union rules — we engaged in some 
teachable moments with administrators. We have needed active and regular consulting with 
department chair and a local leader (e.g., a dean) at each site to negotiate teaching assignments 
and processes for recruitment of faculty as well as implementation of classroom data collection. 
Communication has included in-person meetings, short presentations by project research staff 
at faculty meetings to give information and answer questions about the study, and the creation 
of a 4-page "Administrator's Guide" to the study. The guide became a written document for 
several reasons. One reason was the turnover in administrators over the course of the first year 
in the project, so a written Administrator's Guide was a handy reference for the next new 
person in the job. The guide was also created to address the strong negative reactions to the 
study design by some administrators (chairs and deans), who were sure no instructor would 
agree to be in Group 1. The guide, as revised in response to feedback from project leaders and 
local administrators, has been a worthwhile investment in time as a reference document for 
talking with local administrators. 


Instructors 


One unanticipated challenge of data collection was that the vendor of the online calculus 
readiness pre-test changed their policies and required that researchers obtain anonymized 
student data from instructors (the vendor would not provide it to researchers directly). Thus, 
supports for instructors in order to supply score data were needed. In addition to creating an 


Instructor Guide with timeline of study activities and answers to frequently asked questions, we 
created a three-page illustrated step by step guide for download, anonymizing, and uploading 
score data. These guides, along with links to surveys and logs are organized in a personalized 
Instructor Participant Dashboard (see Figure 2). We asked each instructor for a preferred 
mechanism for communication (email, phone/voice mail, text message). Since the study started 
(January 2018), instructors have received regular reminders to visit their participant dashboard 
to complete study tasks. 
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Figure 2. Screenshot of personalized dashboard for instructors 


Institutional Research Offices 


Another challenge is acquiring, cleaning, and matching of anonymized data for hundreds of 
students across three colleges. From lessons learned on other projects, we met early with staff 
at each of the three Institutional Research offices to plan a test of the data sharing process. As 
of this writing we await test spreadsheets from partner colleges. 


Significance 


In the poster we will share results to date in recruiting of instructor participants, data gathering, 
and testing for data exchange. Tools and tips for establishing clear communication (e.g., the 
Administrator and Instructor Guides, FAQs, and format for meeting with local college leaders) 
can provide foundation for others attempting to work in the public university educational 
research space. 
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Recruitment and Retention of Faculty 


Design 


Tool: Equity in Math Education 


Tool: Relationships 


Tool: Personalized Dashboard 


What: Flipped Instruction. In a "flipped" classroom the 
grounding idea is to switch the traditional uses of in- 
and out-of-class time. Students spend individual 
learning time on lecture-type information and basic skill 
building outside of class. Group learning is the focus of 
class time, with students doing cooperative or 
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(e.g., group work, team-based tasks, labs). 

Where: San José State University, Cal State Los 
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When: Spring ‘18, Fall ‘18, Spring ‘19 

Who: Instructors of Calculus 1, their students, WestEd. 
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Design Visualization — Each Semester 
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Ina TODOS/NCSM position statement, there 
are three necessary conditions to establish just 
and equitable mathematical education for all 
learners: 

1. acknowledge that an unjust social 

system exists. 

2. take action to eliminate inequities and to 
establish effective policies, procedures, and 
practices that ensure just and equitable 
learning opportunities for all, 

3. be accountable by measuring progress so 
changes are made and sustained. 
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Responses, alternatives 
challenging questions. 


Challenging Question: 

1.In what ways might design 
assumptions promote inequity 
and/or equity for participants 
(e.g., faculty, students)? 
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Additional Challenging Questions: 
2. What does it mean for recruitment 
and retention to take action that 

addresses inequity? 

3. How do we know it when we see it 
(i.e., Measure progress)? 

4. How do we include stakeholders? 
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Another Challenging Question: 

5. In what ways does the work in this 
session acknowledge, act, and hold 
us accountable for reducing inequity? 
What is the evidence of it, for you? 


